Here I'm wondering, should we keep the parallel prelude separate? Or pack into the regular ndarray prelude?

It only contains this:

pub mod prelude {
    pub use rayon::prelude::{ParallelIterator, IndexedParallelIterator,
    IntoParallelIterator, IntoParallelRefIterator, IntoParallelRefMutIterator};
}

1. I think it's a good idea to have these traits in a prelude somewhere, we don't want the user to have to extern crate rayon or add rayon as a direct dep if they are purely working in ndarray for a moment.
2. I think we can move these into ndarray::prelude? Conditional on the rayon feature. These will show up/not show up in the prelude depending on the feature flag, but given Rust's rules for glob imports, they only collide if the items are used, it shouldn't actually be an incompatibility. Thoughts?

cc @cuviper Any thoughts about our integration with rayon and our parallel iterators? We're just moving code here and pulling in the rayon 1.0 dep in the main crate, how it's all supposed to be.

I think we can move these into ndarray::prelude? Conditional on the rayon feature.

I lean towards keeping them separate, but putting them in ndarray::prelude would be fine too.

Edit: Earlier, I suggested adding pub use rayon::prelude as parallel to ndarray::prelude, but I no longer think that's a good idea. IMO, if there's a ndarray::parallel module, it would be better to put the parallel prelude at ndarray::parallel::prelude.

Another interesting idea is to reexport all of rayon at the top level of ndarray:

// ndarray/src/lib.rs
pub use rayon;

although users could just depend directly on rayon instead, so I'm not sure if it's worth doing this.

reexporting rayon seems worthwhile.

One other idea for the prelude is to reexport the traits in ndarray::prelude so that their methods can be used easily but hide them from the docs and mangle the trait names so they need to be explicitly imported to use directly:

// ndarray/src/prelude.rs
#[doc(hidden)]
pub use rayon::prelude::{
    ParallelIterator as __rayon_ParallelIterator,
    IndexedParallelIterator as __rayon_IndexedParallelIterator,
    IntoParallelIterator as __rayon_IntoParallelIterator,
    IntoParallelRefIterator as __rayon_IntoParallelRefIterator,
    IntoParallelRefMutIterator as __rayon_IntoParallelRefMutIterator,
};

I think std does this for some traits.

We're merging breaking changes to prepare next major release right?

I think taken together with being an opt-in feature it seems the rayon traits in the regular prelude is a breaking change / method name clash hazard.

I think it's pretty obvious we should have rayon as optional, but maybe I'm wrong?

We're merging breaking changes to prepare next major release right?

Yes, that makes sense to me.

I think it's pretty obvious we should have rayon as optional, but maybe I'm wrong?

I think it's a good idea to make rayon optional for the time being, given the difference in compile times; we can always make it required later.

I think it's good to keep rayon optional even in the future. In my own opinion, it's best not to force certain parallel dependencies on people. Groups can have some very opinionated ideas for how best to parallelize their code. I've especially seen this in the C++ and C realm when you're moving towards the Exascale type of things. You can see very domain specific solutions that aren't obvious that they lead to a noticeable performance difference.

@rcarson3 fair enough, we don't have a plan for transparent/automatic parallelization anywhere though, it's all just separate methods and parallel iterators one can use so far.

we don't have a plan for transparent/automatic parallelization anywhere though, it's all just separate methods and parallel iterators one can use so far.

@bluss I'm aware of this I've had the pleasure of using a few of the parallel iterators to get some easy parallelization out of my code :)

I guess I've become a bit more cautious(?) towards requiring one parallelization approach over another for even the parallelization of iterators/compute kernels as I've become more and more exposed to the different ways people have found to tackle these type of problems for HPC applications.

I'll probably still continue to use this crate no matter what decisions y'all end up going way down the road for the parallel stuff, since I enjoy the ease this crate has made writing numerical code in Rust :)

@rcarson3 Out of curiosity, have you done any HPC-type stuff in Rust? If so, what crates were useful? I have a small SLURM cluster but haven't yet tried running any Rust code across multiple machines.

@jturner314 I haven't gotten the chance to mess around with any HPC stuff in Rust yet. Hopefully, I'll get the chance to experiment with some of this stuff in the next month or so.

Also, I've been eyeballing rsmpi as a traditional method to accomplish HPC type stuff in Rust. It looks like some people from LANL are involved which means we might start seeing Rust used at the DOE national labs which is exciting. I've also been looking at timely-dataflow and differential-dataflow to try and figure out if they could be used in a similar manner to what you might use MPI for.

@rcarson3 Thanks for the links. The dataflow projects look particularly exciting, and relatively easy-to-use.

This should be ready. Using separate parallel prelude, and it contains the rayon traits and the par_azip macro.

I think the bounds on the &mut T producing parallel iterators are too strict. They currently use T: Send + Sync but for example a corresponding IntoParallelIterator on &mut [T] in rayon only requires T: Send.

@bluss

Any thoughts about our integration with rayon and our parallel iterators?

Looks good to me!

I noticed that views and zips are unindexed -- AFAIK their length is known, so is this just because they're hard to split at arbitrary indexes? (I have the same limitation in rayon-hash.)

I think the bounds on the &mut T producing parallel iterators are too strict. They currently use T: Send + Sync but for example a corresponding IntoParallelIterator on &mut [T] in rayon only requires T: Send.

Is there a technical reason why you're requiring Sync? Or just an oversight?

@cuviper They are impossible to split at arbitrary linear indexes, if they have more than one dimension. i.e a 3 × 3 × 3 array view should be split along a good combination of axis and index.

For example along axis 0, index 1 (That split produces the following pieces: view of 1 × 3 × 3 and 2 × 3 × 3 see also illustrations).

And we want to split them in a way that preserves element locality. This is a dynamic property. Maybe the elements are closest to each other along axis 1 and furthest from each other along axis 0, then we split along axis 0. We want to increase our chances to end up with a contiguous chunk to work on for the final single job.

We can split successively on different axes, for example when the first axis has been split down to length 1, we go on to the current axis of furthest separation with len > 1 and split along that.

What can be added is to make more producers indexed, in their 1D cases only, but I'm unsure if that's possible. Say we have an Parallel<ArrayView<f64, D>> and when D = Ix1 we will additionally implement IndexedParallelIterator. Since the parallel traits cooperate a bit, I'm not sure if that works out.

I think the Sync thing is just an oversight / “old quick solution to be safe”, I don't remember any reason we require that.

we want to split them in a way that preserves element locality.

I see -- and I suppose direct Array access can just split the backing storage, regardless of the formal shape? If that's the case, then it seems like you could do the same for iterating &Array and &mut Array, rather than going through views where that contiguity is no longer guaranteed.

edit: I think I mistook the macros for the indexed AxisIter part as a direct Array. Do you not have IntoParallelIterator for Array at all?

What can be added is to make more producers indexed, in their 1D cases only, but I'm unsure if that's possible. Say we have an Parallel<ArrayView<f64, D>> and when D = Ix1 we will additionally implement IndexedParallelIterator. Since the parallel traits cooperate a bit, I'm not sure if that works out.

The opt_len hack might be hard without specialization, unless you separately implemented each possible D for ParallelIterator, but that's just an optimization for collect to act like collect_vec.

I think it would work fine to just implement IndexedParallelIterator for the 1D case, and leave ParallelIterator as a more generic unindexed implementation. If that doesn't work for some reason, I'm willing to consider that as a rayon bug (whether or not it's possible to fix).

@cuviper We have parallel iterators for array views, for &Array and &mut Array, but not Array itself. Just like we don't have a regular by value iterator.

Implementing Indexed parallel iterators is left for later. This PR is mostly moving the code from a separate crate into the main crate.

We have parallel iterators for array views, for &Array and &mut Array, but not Array itself. Just like we don't have a regular by value iterator.

Implementing Indexed parallel iterators is left for later.

OK, but note that having <&Array as IntoParallelIterator>::Iter = Parallel<ArrayView> will restrict what you can do at the type level. Using something like Iter = Parallel<&Array> could still be implemented in terms of the view-style Producer from drive_unindexed, but would leave you the flexibility to switch to an indexed implementation later (without a breaking change).

An ArrayView is to an Array like a slice to a Vec, so I think it makes sense to implement the indexed parallel iterator on the arrayview, we don't have anything separate from array views that handles an &Array.

Right, but if I understand your earlier comment about preserving locality, a general ArrayView is not necessarily contiguous. Whereas a full &Array would be contiguous, and could more easily support arbitrary splits, no?

All I'm suggesting is:

// (with lots of omissions...)
impl IntoParallelIterator for &Array {
    type Iter = Parallel<&Array>;
}
impl ParallelIterator for Parallel<&Array> {
    fn drive_unindexed<C>(self, consumer: C) -> C::Result {
        // A delayed view conversion lets you do something else later
        self.0.view().into_par_iter().drive_unindexed(consumer)
    }
}

Aha that makes sense. We'll have to look if we can use that for extra features. In general, arrays and views have about the same properties in everything, including supporting custom strided axes, so I'm not sure if I'm right about not grasping the concept.